{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.5"},"colab":{"name":"2.3 Keras Tooling.ipynb","provenance":[{"file_id":"1sP5ILsdUQlbzPCOlPQpSd1Zw0fZ2xyX5","timestamp":1573397192284}],"collapsed_sections":[]}},"cells":[{"cell_type":"markdown","metadata":{"id":"71bIHkK_PxZN","colab_type":"text"},"source":["## Introduction\n","\n","In this notebook, we will:\n","\n","1. Get more practice manipulating our training data to fit our neural networks\n","2. Learn how to use some of the tooling in the Keras ecosystem\n","3. Learn how to debug issues with our neural networks\n"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-01-13T21:10:06.123871Z","start_time":"2019-01-13T21:10:00.647687Z"},"id":"Sn5PZP-3PxZY","colab_type":"code","outputId":"e5a04b90-e3d4-493b-896b-96a9505204dc","executionInfo":{"status":"error","timestamp":1572717151126,"user_tz":240,"elapsed":4593,"user":{"displayName":"Brian","photoUrl":"https://lh4.googleusercontent.com/-y5a64nLFbc4/AAAAAAAAAAI/AAAAAAAAAXs/4RRMVlUcGtY/s64/photo.jpg","userId":"11680685692091329827"}},"colab":{"base_uri":"https://localhost:8080/","height":405}},"source":["%tensorflow_version 1.x\n","import numpy as np\n","\n","from keras.models import Sequential\n","from keras.layers import Dense, Activation\n","from keras.datasets import mnist\n","from keras.optimizers import Adam\n","from keras.utils import to_categorical\n","from keras.callbacks import TensorBoard\n","\n","from keras_tqdm import TQDMNotebookCallback"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Using TensorFlow backend.\n"],"name":"stderr"},{"output_type":"display_data","data":{"text/html":["<p style=\"color: red;\">\n","The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n","We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n","or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n","<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{"tags":[]}},{"output_type":"error","ename":"ModuleNotFoundError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)","\u001b[0;32m<ipython-input-1-39f2c8fb61fb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallbacks\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mTensorBoard\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mkeras_tqdm\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mTQDMNotebookCallback\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m","\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'keras_tqdm'","","\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"]}]},{"cell_type":"markdown","metadata":{"id":"QYmFap-UPxZj","colab_type":"text"},"source":["## Load Dataset"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-01-13T21:10:06.675091Z","start_time":"2019-01-13T21:10:06.127257Z"},"id":"XSTqSmUDPxZl","colab_type":"code","colab":{}},"source":["# Load Datset\n","(x_train, y_train), (x_test, y_test) = mnist.load_data()\n","\n","print(\"Training Dataset Size: \", x_train.shape)\n","print(\"Training Labels Size: \", y_train.shape)\n","print(\"Testing Dataset Size: \", x_test.shape)\n","print(\"Testing Labels Size: \", y_test.shape)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"syBhXZxiPxZs","colab_type":"text"},"source":["### Preparing the Data\n","\n","1. Use the `numpy.resize()` API to transform our input data from a `28x28` image to a `784` 1-D array. See: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.resize.html\n","2. Each pixel value is an integer between $[0, 255]$.  Scale the input to be in range of $[0, 1]$ by dividing by `255`.\n","3. Use the `keras.utils.to_categorical` API to convert our digit labels from integers (0, 1, 2,..., 9) to a one-hot encoding format.  See: https://keras.io/utils/#to_categorical"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-01-13T21:10:06.687093Z","start_time":"2019-01-13T21:10:06.678024Z"},"id":"LLBmhBJvPxZu","colab_type":"code","colab":{}},"source":["# Resize image to 1D array and values to between [0, 1]\n","# x_train = ...\n","# x_test = ...\n","\n","# Convert our labels to one-hot encoded format\n","# y_train = ...\n","# y_test = ...\n","\n","\n","print(\"Training Dataset Size: \", x_train.shape)\n","print(\"Training Labels Size: \", y_train.shape)\n","print(\"Testing Dataset Size: \", x_test.shape)\n","print(\"Testing Labels Size: \", y_test.shape)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"CSG6ZTZ4PxZ0","colab_type":"text"},"source":["## Defining Our Neural Network "]},{"cell_type":"markdown","metadata":{"id":"-vrGbXNtPxZ2","colab_type":"text"},"source":["Network specificatoins:\n","\n","* `3` hidden layer neural network\n","* For each hidden layer `50` hidden units with the `relu` activation function\n","* Define an output layer with the appropriate activation and output units\n","* Use an appropriate loss function for categorical data\n","* Use the `Adam` optimizer with a learning rate of `0.01`\n","* Add the `accuracy` metric to our training. See: https://keras.io/metrics/"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2018-09-14T00:20:11.060971Z","start_time":"2018-09-14T00:20:10.971739Z"},"id":"vJfCtKdXPxZ4","colab_type":"code","colab":{}},"source":["# Define our neural network\n","model = Sequential()\n","\n","# Add your neural network layers here\n","# model.add(...)\n","# ...\n","# model.summary()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2018-09-14T00:20:11.125930Z","start_time":"2018-09-14T00:20:11.066348Z"},"id":"JuLlMY6sPxZ9","colab_type":"code","colab":{}},"source":["# Compile your model and define your loss and optimization algorithm\n","# model.compile(...)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"fVQ2Cm8wPxaC","colab_type":"text"},"source":["## Training our Neural Network with Additional Tooling\n","\n","* `batch_size` of `100`\n","* `epoch` of `5`\n","* 15% validation split\n","\n","Add the following additonal callbacks:\n","\n","* TQDM Keras callback: gives us a much nicer visualization of the progress in our notebook environment.  Make sure you pass `verbose=0` to `fit()` so we don't get two outputs.  See: https://github.com/bstriner/keras-tqdm\n","* Tensorboard callback: Records data about training so we can visualize it in the TensorBoard tool.  Use the following settings: `log_dir=./logs_2.3, histogram_freq=1, batch_size=32, write_graph=True, write_grads=True, write_images=True`)  See: https://keras.io/callbacks/ and https://www.tensorflow.org/guide/summaries_and_tensorboard\n"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2018-09-14T00:21:35.464730Z","start_time":"2018-09-14T00:20:11.129572Z"},"id":"lBl0KEHuPxaE","colab_type":"code","colab":{}},"source":["# Fit your model on the training data\n","# callbacks = []\n","# history = model.fit(..., callbacks=callbacks)\n","# history.history"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"gZ1QwD2wPxaN","colab_type":"text"},"source":["## Viewing Results in Tensorboard\n","\n","To view results in Tensorboard, run the following in the command line:\n","\n","```\n","> tensorboard --logdir=<logdir>\n","```\n","\n","where `<logdir>` is the directory specified above in the Tensorboard call (in this case `logs_2.3/`)"]},{"cell_type":"markdown","metadata":{"id":"G5RPRoZzPxaQ","colab_type":"text"},"source":["## Questions\n","\n","1. Look at Tensorboard's `scalars` tab.  Are we overfitting the model?\n","2. Look at the `Graphs` tab.  Our `model.summary()` only shows four layers above.  Why are there so many more blocks in this diagram of our neural network?\n","3. Look at the `Histograms` and `Distributions` tab.  What are they showing us?  How can this help use debug our neural network?"]},{"cell_type":"markdown","metadata":{"ExecuteTime":{"end_time":"2018-09-10T23:22:18.273507Z","start_time":"2018-09-10T23:22:18.270129Z"},"id":"LLWGHLt1PxaR","colab_type":"text"},"source":["## Summary\n","\n","* Keras callbacks allow you to record metrics (and customize the behavior) during training.\n","* Tensorboard is a useful tool for the Tensorflow backend to visualize and debug your networks.\n","* These tools are essential when building your own networks because of all the debugging and hyperparameter tuning that is needed."]},{"cell_type":"code","metadata":{"id":"chwKrp0cPxaT","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}